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Automated Verification of User Interface Tests on 
Low-End Emulators and Devices 



BACKGROUND OF THE INVENTION 

1. Field of ^he Inven'tlon. 

[0001] The present, invention relates generally to hard- 

ware and software testing and verification, and specifically to 
testing software on low-end emulators and computing devices- 

2. Descrip-tion of the Related Art. 

[0002] The meanings of acronyms and certain terminology 

used herein are given in Table 1: 



Table 1 



API 


Application programming interface 


CLDC 


Connected, limited device configuration. CLDC 
is suitable for devices with 16/32-bit 
RISC /CISC microprocessors /controllers , having 
as little as 160 KB of total memory available. 


HTTP 


HyperText Transfer Protocol 


IP 


Internet Protocol 


J2EE 


Java 2 Enterprise Edition 


J2ME 


Java 2 Micro Edition 


J2SE 


Java 2 Standard Edition 


JAD 


Java application descriptor 


JAR - , 


Java archive. " . . 


JDTS 


Java Device Test Suite 


MIDlet 


A MIDP application 


MIDP 


Mobile information device profile. A set of 
Java APIs, . .which, together with the CLDC, 
provides a complete J2ME application runtime 
environment targeted at mobile information 
devices. 


MIDP-RI 


MIDP-Ref erence Implementation 


UI 


User Interface 
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WTK 



Wireless Toolkit 



[0003] MIDP is defined in Mobile Information Device 

Profile {JSR-37), JCP Specification, Java 2 Platform, Micro 
Edition, 1.0a (Sun Microsystems Inc., Palo Alto, California, 
5 December 2000). MIDP builds on the Connected Limited Device 
Configuration (CLDC) of the Java 2 Platform, Micro Edition 
(J2ME) (available from Sun Microsystems Inc., Palo Alto, Cali- 
fornia) . The terms Sun, Sun Microsystems, Java, J2EE, J2ME, 
J2SE, and the Sun logo are trademarks or registered trademarks 

10 of Sun Microsystems, Inc. in the United States of America and 
other countries. All other company and product names may be 
trademarks of their respective companies. A portion of the dis- 
closure of this patent document contains material that is sub- 
ject to copyright protection. The copyright owner has no objec- 

15 tion to the facsimile reproduction by anyone of the patent 
document or the patent disclosure, as it appears in the Patent 
and Trademark Office patent file or records, but otherwise re- 
serves all copyright rights whatsoever. 

[0004] Tools have been developed in recent years to aid 

20 in the design verification of hardware and software systems, 
for- example software suites, hardware circuitry, and programma- 
ble logic designs. In order to assure that the design complies 
with its specifications, it is common to generate a large num- 
ber of input or instruction sequences to assure that the design 

25 operates as intended under a wide variety of circumstances. In 
general, test systems produce a report indicating whether tests 
have been passed or failed, and, in some cases may even indi- 
cate a module that is estimated to be faulty. 

[0005] Conventionally, in order to test a device under 

30 development (such as a mobile information device) , or to test 
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software designed to run on such a device, a developer connects 
the device to an appropriate test system. The target device un- 
der test may be connected to the test system either directly or 
via a communication emulator. The developer selects a battery 
of test programs to run on the target device while monitoring 
its behavior. Running the complete battery of tests can com- 
monly take many hours or even days. This problem is particu- 
larly acute in testing low-end computing devices, such as cel- 
lular telephones and other mobile information devices, which 
have limited computing power and memory resources. 

[0006] It is common to execute tests using a user in- 

terface (UI) , which is typically a graphical user interface. 
This interface is used to control execution of the tests, and 
to verify that what appears on the device screen under certain 
circumstances is correct and consistent. Tests, which are in- 
teractively verified in this manner, are referred to herein as 
^'UI tests''. However, since the number of different tests need 
for a target device under development today can easily ex- 
ceed 800 or more, running such tests using a conventional user 
interface is indeed an onerous undertaking. It has been found 
that the sheer number of results to be evaluated is likely to 
incur operator errors during the verification process. Further- 
more, running such tests using a user interface can consume 
valuable time and can reduce the productivity of an engineering 
group. Thus, testing the target device can become a serious 
bottleneck in the development cycle. 

SUMMARY OF THE INVENTION 

[0007] The invention provides a level of automation for 

interactive tests similar to what exists today for non- 
interactive tests. A complete test cycle involving both non- 
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interactive and interactive tests can be completed with minimal 
input from an operator or test engineer. 

[0008] In accordance with embodiments of the present 

invention one or more target devices are connected to a test 
5 server, either directly or via an emulator. Whenever a device 
completes a test or a bundle of tests, it reports the results 
to the server and requests the next bundle of tests to execute. 
Based on the report, the server selects the next bundle of 
tests to assign to that device. 

10 [0009] In order to improve testing efficiency, the 

testing process is automated according to embodiments of the 
present invention by recording each of the tests once, captur- 
ing the user interface operations as a record, and then rerun- 
ning the tests repetitively and automatically on the same or 

15 different instances of the target device, substantially without 
human intervention . 

[0010] The invention provides a method for testing com- 

puting devices, which is carried out in a first phase of opera- 
tion by downloading a test program a first time from a server 

20 for execution by a first computing device coupled thereto, exe- 
cuting the test program on the first computing device to • pro- 
duce program events, recording the program events, and captur- 
ing first screens of the first computing device that are dis- 
played responsively to the program events while the test pro- 

25 gram is executing on the first computing device. The method is 
carried out in a second phase of operation by downloading the 
test program a second time for execution by a second computing 
device, replaying the test program on the . second computing de- 
vice to reproduce the program events, capturing second screens 

30 of the second computing device that are- displayed while execut- 
ing the test program on the second computing device respon- 
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sively to the reproduced program events^ and comparing at least 
one of the first screens to a corresponding one of the second 
screens . 

[0011] According to one aspect of the method, the sec- 

5 ond phase of operation is performed automatically and substan- 
tially without human intervention. . 

[0012] According to another aspect of the method, the 

second phase of operation includes automatically injecting the 
program events into an event handler. 
10 [0013] According to an additional aspect of the method, 

the program events are injected into the event handler as ex- 
ecutable code. 

[0014] According to one aspect of the method, the pro- 

gram events are injected into the event handler as a stream of 
15 event code for processing by the event handler. 

[0015] In another aspect of the method, the steps of 

replaying, capturing second screens and comparing are performed 
automatically and substantially without human intervention. 

[0016] According to a further aspect of the method, the 

20 program events comprise a time interval between a current event 
and another event. 

[0017] According to yet another aspect of the method, 

the program events comprise program actions. 

[0018] According to still another aspect of the method, 

25 the program events comprise user actions. 

[0019] An additional aspect of the method downloading 

the test program a first time and downloading the test program 
a second time are performed by downloading a JAR file and a JAD 
file. 

30 [0020] According to one aspect of the method, the sec- 

ond computing device includes a plurality of second computing 
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devices, and the second phase of operation is performed concur- 
rently on the plurality of second. computing devices. 

[0021] According to another aspect of the method, the 

second computing device and the test program are MIDP compli- 
5 ant . 

[0022] The invention provides a computer software prod- 

uct, including a computer-readable medium in which computer 
program instructions are stored, which instructions, when read 
by a computer, cause the computer to perform a method for test- 

10 ing computing devices, which is carried out in a first phase of 
operation by downloading a test program a first time from a 
server for execution by a first computing device coupled 
thereto, executing the test program on the first computing de- 
vice to produce program events, recording the program events, 

15 and capturing first screens of the first computing device that 
are displayed responsively to the program events while the test 
program is executing on the first computing device. The method 
is carried out in a second phase of operation by downloading 
the test program a second time for execution by a second com- 

20 puting device, replaying the test program on the second comput- 
ing device- to reproduce the program events, capturing second 
screens of the second computing device that are displayed while 
executing the test program on the second computing device re- 
sponsively to the reproduced program events, and comparing at 

25 least one of the first screens to a corresponding one of the 
second screens. 

[0023] The invention provides a system for testing com- 

puting devices coupled to a server having a test framework exe- 
cuting therein that is adapted for interaction with the. comput- 
30 ing devices, wherein in a first phase of operation a test pro- 
gram is executed by a first computing device coupled thereto to 
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produce program events. The server is adapted to record the 
program events, and to capture first screens of the first com- 
puting device that are displayed responsively to the program 
events. In a second phase of operation, the test program is 
5 executed by a second computing device coupled to the server un- 
der control thereof to reproduce the program events. The server 
is adapted to capture second screens of the second computing 
device that are displayed responsively to the reproduced pro- 
gram events, and to compare at least one of the first screens 
10 to a corresponding one of the second screens. 

^ BRIEF DESCRIPTION OF THE DRAWINGS 

[0024] For a better understanding of the present inven- 

tion, reference is made to the detailed description of the in- 
vention, by way of example, which is to be read in conjunction 

15 with the following drawings, wherein like elements are given 
like reference numerals, and wherein: 

[0025] Fig. 1 is a block diagram that schematically il- 

lustrates a system for automatically testing one or more target 
devices, in accordance with an embodiment of the present inven- 

20 tion; 

[0026] Fig. 2 is a block diagram that schematically il- 

lustrates software program components running on a test 
server and target devices in accordance with a disclosed em- 
bodiment of the present invention; 
25 [0027] Fig. 3 is a flow chart illustrating a recording 

phase of a method of automated verification of user interface 
tests in accordance with a disclosed embodiment of the inven- 
tion; 

[0028] Fig. 4 is a flow chart illustrating a replay 

30 phase of a method of automated verification of user interface 



P9340 



47856 



Ver. 47856S6.doc 



tests in accordance with a disclosed embodiment of the inven- 
tion; 

[0029] Fig. 5 is a fragmentary . schematic illustration 

of a series of byte arrays representing a test record from an 
5 execution in recording mode in accordance with a disclosed em- 
bodiment of the invention; 

[0030] Fig. 6 is a fragmentary schematic illustration 

of a series of byte arrays representing an updated test record 
from an execution in an updating mode in accordance with a dis- 
10 closed embodiment of the invention; and 

[0031] Fig. 7 is a flow chart illustrating a method of 

updating a test record in accordance with a disclosed . embodi- 
ment of the invention. 

DETAILED DESCRIPTION OF THE INVENTION 

15 [0032] In the following description, numerous specific 

details are set forth in order to provide a thorough under- 
standing of the present invention. It will be apparent to one 
skilled in the art, however, that the present invention may be 
practiced without these specific details. In other instances 

20 well-known circuits, control logic, and the details of computer 
-program instructions for conventional algorithms and processes 
have not been shown in detail in order not to unnecessarily ob- 
scure the present invention. 

[0033] Software programming code, which embodies as- 

25 pects of the present invention, is typically maintained in per- 
manent storage, such as a computer readable medium. In a cli- 
ent-server environment, such software programming code may be 
stored on a client or a server. The software programming code 
may be embodied on any of a variety of known media for use with 

30 a data processing system. This includes, but is not limited to. 
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magnetic and optical storage devices such as disk drives, mag- 
netic tape, compact discs (CD's), digital video discs (DVD's), 
and computer instruction signals embodied in a transmission me- 
dium with or without a carrier wave upon which the signals are 
modulated. For example, the transmission medium may include a 
communications network, such as the Internet . 

[0034] Current embodiments of the invention have been 

adapted to particular platforms, in particular MIDP-RI 2.0 and 
J2ME Wireless Toolkit 2.0 Beta 2 or higher, both available from 
Sun Microsystems, Inc. However, the invention is not limited to 
these platforms, and the, teachings herein may be readily 
adapted, using ordinary skill in the art, to many other plat- 
forms by implementing automation-related API's or their equiva- 
lents. 

15 Sys'bem Architecbure . 

[0035] Reference is now made to Fig. 1, which is a 

block diagram that schematically illustrates a system 10 for 
automatically testing one or more target devices 12, in accor- 
dance with an embodiment of the present invention. The sys- 

20 tem 10 is built around a test server 14, which is described in 
greater detail hereinbelow. The target devices 12 are typically 
low-end devices, with limited computing power and memory, for 
example, cellular telephones or personal digital assistants 
(PDA's). In the description that follows, the target devices 12 

25 are assumed to comply with MIDP, but the principles of the pre- 
sent invention are equally applicable to other types of low-end 
computing devices, operating in accordance with other standards 
and specifications. The test server 14 typically comprises a 
programmable processor, and has suitable communication inter- 
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faces, such as wireless or wired interfaces, for communicating 
with the target devices 12 simultaneously. 

[0036] Optionally, the test server 14 communicates with 

at least some of the target devices 12 through a communications 
5 bridge 16. The communications bridge 16 links the test 
server 14 and the target devices 12, and, for example could 
emulate a radio interface of the target devices 12. The commu- 
nications bridge 16 is typically used for devices that are rig- 
idly programmed so that they cannot directly communicate with 

10 the test server 14. For example, the devices may all be pro- 
vided with the same IP address. Multiple communications bridges 
of this sort may be connected to the test server 14 in paral- 
lel, but only one is shown in Fig. 1 for the sake of simplic- 
ity. The communications bridge 16 has multiple ports, for con- 

15 necting to multiple target devices 12 for concurrent operations 
therewith. On the side of. the test server 14, however, the com- 
munications bridge 16 typically has only a single IP address. 
Therefore, in this embodiment, the IP address cannot be used 
conveniently to identify the individual target devices 12, and 

20 an alternative ID is typically used, as disclosed in described 
in commonly assigned Application No. (STC File No. 47979), en- 
titled Parallel Text Execution on Low-End Emulators and De- 
vices, which is herein incorporated by reference. 

[0037] A user 18 of the system 10, such as a test engi- 

25 neer, interacts with the software elements of the test 
server 14, during certain phases of operation, as described in 
further detail hereinbelow. The user 18 operates a work- 
station 20 and interacts with the test server 14 via a user in- 
terface that is presented on a display 22 or a display 24 of 

30 the target devices 12. 



P9340 



47856 



Ver. 47856S6.doc 



11 

[0038] In configurations of the system 10 in which more 

than one target device is being tested simultaneously, each of 
the target devices 12 receives a unique identifier (ID) for 
communicating with the test server 14. Typically, the identi- 
5 fier may comprise a unique Internet Protocol (IP) address that 
is assigned to each of the target devices 12 for communicating 
with the test server 14. Alternatively, the test server 14 may 
assign identifiers of other types, or the identifiers may be 
pre-stored in the target devices 12. Suitable methods for as- 

10 signing and using these identifiers are described in the above- 
noted Application No. (STC File -No. 47979). 

[0039] Reference is now made to Fig. 2, which is a 

block diagram that schematically illustrates software program 
components running on the test server 14 and the target de- 

15 vices 12 (Fig. 1), in accordance with a disclosed embodiment of 
the present invention. Elements of this software may be pro- 
vided to the test server 14 and the target devices 12 on tangi- 
ble media, such as optical or magnetic storage media or semi- 
conductor memory chips. Alternatively, the software may be 

20 downloaded to the test server 14 and/or to the target de- 
vices 12 in electronic form, over a network or over the air, 
for example. 

[0040] The test server 14 includes a test framework 26, 

which generates and deploys the tests to be carried out by the 

25 target devices 12. The tests typically are packaged in the form 
of Java applications contained in a set of JAD and JAR files. 
Each JAR file of this sort, together with its accompanying JAD 
file, is referred to hereinbelow as a test bundle 28. A suit- 
able test framework for this purpose is described, for example, 

30 in commonly assigned Application No. (STC File No 47900), enti- 
tled. Automated Test Execution Framework with Central Manage- 
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menty which is herein incorporated by reference. The user 18 
(Fig. 1) initially selects the tests to be executed. Alterna- 
tively, other test frameworks may be used for generating re- 
quired test files, as will be apparent to those skilled in the 
5 art . 

[0041] A test manager 30 in the test server 14 is re- 

sponsible for distributing the tests among target devices 12, 
based on the unique client identifiers mentioned above. Typi- 
cally, whenever one of target devices 12 requests a new test 

10 bundle, the test manager 30 reads the request and assigns a new 
thread 32 to handle it. This thread 32 is then responsible for 
the particular request. In general, a different thread 32 is 
created for each request involving processing and communication 
with the target devices 12. For example, a request for a new 

15 test bundle, for a new test within a current test bundle, or a 
request for results are all implemented by the creation of a 
different thread 32. After assigning the thread 32, the main 
thread of the test manager 30 waits for the next client re- 
quest. This arrangement, together with the unique client iden- 

20 tifier, ensures that the test server 14 will be able to handle 
multiple target devices 12 simultaneously without confusion. 

[0042] In order to run Java applications, the target 

devices 12 are MIDP compliant and each contain an implementa- 
tion of the Connected Limited Device Configuration specifica- 

25 tion, CLDC 34, with an implementation of the Mobile Information 
Device Profile specification, MIDP 36 running over CLDC 34. The 
applications that run on this technology, such as the tests 
supplied by the test framework 26, are known as MIDlets. These 
applications are created by extending an API MIDlet class of 

30 MIDP 36. Thus, each test bundle 28 is actually a MIDlet, pack- 
aged in the form of a JAD/JAR file pair. 
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[0043] The test bundle 28 is typically downloaded to 

the target devices 12 in a two-step process: 

[0044] 1. The test server 14 downloads the JAD file, 

which contains environment settings and some environment de- 
5 mands . Application Manager Software, referenced as AMS 38, 
which is typically a part of a browser built into the target 
devices 12, evaluates the JAD file to ensure that the device is 
able to accept the MIDlet. For example, the JAD file for a 
given MIDlet may specify that the device must support MIDP ver- 

10 sion 2.0 or higher. Otherwise, the device would reject the ap- 
plication download, and would thus save the time that would be 
consumed by downloading the much larger JAR file. 

[0045] 2. After completing all the relevant checks, the 

AMS 38 reads from the JAD file the location of the correspond- 

15 ing JAR file on the test server 14 and asks to download the JAR 
file to one or more of the target devices 12. The JAR file con- 
tains all the relevant classes of the test bundle 28. 

[0046] Once the JAR file for a given test bundle is 

downloaded to one of the target devices 12 and stored in the 

20 local device memory, the device is ready to run the tests in 
the bundle. Every JAR file that the AMS 38 downloads to the 
target devices 12 typically contains an agent 40, which is used 
to run the tests, in addition to the classes corresponding to 
the tests themselves. To start test execution the AMS 38 runs 

25 the agent class. The agent 40 then addresses the test server 14 
in order to receive instructions regarding the next test to run 
(getNextTest ) and to report test results (sendTestResult ) , 
typically using a protocol based on HTTP. Each test in the test 
bundle 28 may correspond to a respective class in the JAR file. 

30 Alternatively, a class may include several tests, which invoke 
multiple test cases within the class. Each client request that 
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is addressed by the agent 40 to the test server 14 includes the 

unique ID that has been assigned to the particular one of the 
target devices 12, so that the test server 14 is able to recog- 
nize the client and serve it in the correct manner. 

5 Implemexi'ba'bion . 

[0047] In one embodiment^ the test framework 26 

(Fig. 2) has been implemented as a modification of the test 
framework^ ^^Java Device Test Suite" (JDTS) (version 1.0 or 
higher), available from Sun Microsystems^ Inc., which employs 

10 MIDP. A Java interface is constructed to artificially inject 
events into the MIDP event handler of the target devices 12 or 
the communications bridge 16. This interface enhances the Java 
Device Test Suite by providing a programmatic capability to 
control the user interface as if it were being done interac- 

15 tively via the keyboard or mouse. 

[0048] Artificial events may be injected into the na- 

tive event code, and could be pushed onto the event stream from 
which MIDP reads its events. Alternatively, artificial events 
may be injected directly into Java code within the Java event 

20 handler of MIDP. The latter approach has several advantages 
over the former. Events include user actions, such as mouse 
clicks or selection, keyboard operation, and various program 
actions, e.g., I/O operations, program exceptions, time-outs,, 
and initiation and completion of operations . 

25 [0049] The Java event handler of MIDP is instantiated 

at runtime via a system property setting, as shown in 
Listing 1. For example, by modifying the configuration property 
com. sun. Icdui . eventHandler, a MIDP build can be modified as to 
which event handler is used at runtime. This may be done with- 

30 out any special compilation options being set, and without re- 
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compilation of the event handler for test automation purposes. 
Furthermore, the code used to inject events artificially is 
separated from the remainder of the MIDP bundle by encapsula- 
tion in a separate subclass of the default event handler.. Ac- 
5 cordingly, the native code remains unchanged across all MIDP 
platforms and ports. The modified Java event handler is usable 
in all MIDP implementations, which use the base Java event han- 
dler functionality. Advantageously, the user interface automa- 
tion functionality can be cleanly removed, not only from the 

10 binary version of the software, but also from its source code, 
simply by removing the separate event handler subclass and its 
related interfaces. Other techniques known to the art for im- 
plementing the event handler may also be employed. 

[0050] The programmatic interface for generating events 

15 is only part of the modification of the Java Device Test Suite 
disclosed herein. If merely an interface to generate events 
were provided in the Java Device Test Suite, the UI tests would 
have to be manually converted into Java test files, and the 
event sequence in each UI test encoded in the source code as 

20 calls to the event generation interface. Furthermore, time in- 
tervals between events would also need to be encoded. Preparing 
such Java programs would be time consuming, would require 
trained programmers, and would involve extensive debugging. 

[0051] A ^^record" feature is provided in the modified 

25 event handler. Using this feature, the operator is involved 
only to the extent of running the interactive UI tests one time 
in a recording mode, while a reference test device is linked to 
a server running the Java Device Test Suite. During this one- 
time run, test event sequences, time intervals between events, 

30 and captured screens are recorded. In some embodiments, the 
time intervals between successive events may be recorded. Al- 
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ternatively, each event may be time- stamped, and the inter- 
event time intervals computed subsequently. In any case, all 
recorded information is stored in separate files, independently 
of the corresponding UI tests. Subsequently, when connected to 
5 a device-under-test having the same specification as the device 
used for recording the information, the Java Device Test Suite 
restores the saved event sequence, and passes it to the modi- 
fied event handler for automatic replay, substantially without 
human intervention. A screen capture taken during the replay, 

10 which represents the response of the device-under-test, should 
match that taken at the end of the manually run interactive UI 
test using the reference test device. Furthermore, all the in- 
termediate screens that are generated by the device-under-test 
in response to stimulation by the Java Device Test Suite should 

15 match corresponding intermediate screens that were captured 
when the test was initially recorded by the operator. 
ScreenGrabber . 

[0052] Event generation API's of the modified event 

handler are dependent upon the ^'screen grabber" functionality 

20 that is already present in MIDP (ver. 1.0.3 or higher). In or- 
der to facilitate understanding of the invention, the screen 
grabber functionality is explained briefly. The public class 
ScreenGrabber is a class that returns a hash of the pixels on 
the MIDP display drawing region. The principal method of the 

25 class ScreenGrabber is a method getData, which returns a digest 
of the ^^pixelmap" of the current MIDP display area. Currently, 
the pixelmap is defined as the drawing area, as well as the 
status bar at the top of the screen and the area of the screen 
devoted to scroll arrows and softkey menu labels. 

30 [0053] A method getlnstance returns a reference to an 

instance of the class ScreenGrabber. 
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Event: Sequence. 

[0054] A class Event Sequence encapsulates the semantics 

of a single MIDP events representing a sequence of events and a 
screen capture taken at the end of those events. 
5 [0055] A first class constructor EventSequence recre- 

ates a stored event sequence and its associated screen capture 
resulting from the input stream. 

[0056] A second class constructor EventSequence creates 

a new EventSequence object based on a set of other EventSe- 

10 quence objects. This produces a composite EventSequence object, 
which is treated as a sequence by an AutomatedHandler interface 
(described hereinbelow) during testing. That is, each subse- 
quence and its associated capture screen capture are replayed. 
If any subsequence fails, the value false is returned. If all 

15 of the subsequences pass, the value true is returned. The class 
EventSequence has the following methods. 

[0057] A method appendSequence appends an EventSequence 

object to another EventSequence object, and thereby creates a 
composite event sequence. 

20 [0058] A method toByteArray writes an EventSequence ob- 

ject and its corresponding screen capture to a designated out- 
put stream. The EventSequence object is represented as an array 
of bytes, which can be passed to the class constructor to re- 
create the event sequence. 

25 Au*toma't±on Handler. 

[0059] An abstract automation handler interface pro- 

vides an automation handler class, which includes options to 
record and playback a sequence of events for automation pur- 
poses, and to capture screen contents. It has the following 

30 methods. 
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[0060] A method captureScreen captures the current con- 

tents of the physical display in the form of a byte array. The 
byte array may be a reduced form of the display, such as a 
checksum or hash, but it is guaranteed to be unique for the 
5 display, such that no two differing displays can have the same 
return value. 

[0061] A method registerHotKey establishes a specific 

key code for the given action to serve as a ^^hotkey''. 

[0062] A method registerSequenceHandler establishes a 

10 sequence handler to handle the completion of event sequences 
and screen captures that occur as a result of a hotkey press. 
Once a sequence handler is established, that sequence handler 
is called whenever an event sequence is completed through invo- 
cation of a method stopEventSequence, which is disclosed here- 

15. inbelow. The sequence handler may also be invoked as a result 
of a hotkey being pressed, which the event handler recognizes 
as a signal to stop event recording. An Event Sequence object 
passed to the sequence handler is the same as the return result 
of the StopEventSequence method. If a sequence handler has been 

20 set, and the method registerSequenceHandler is invoked again, 
the existing sequence handler is discarded and the new sequence 
handler used. If the value null is passed to the method regis- 
terSequenceHandler, any previous sequence handler is discarded. 
[0063] A method replayEventSequence replays a given 

25 event sequence, which is represented in an EventSequence ob- 
ject. It captures the contents of the screen that is displayed 
at the end of the event sequence, and compares it to a captured 
screen that is included in the EventSequence object. 

[0064] A method startEventSequence initiates recording 

30 of an event sequence. Any event sequence currently being re- 
corded is destroyed and a new event sequence started. 
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[0065] A method stopEventSequence stops recording of an 

event sequence, and captures the current contents of the 
screen. It then returns an EventSequence object, which is the 
representation of the entire event sequence as well as the 
5 screen capture resulting at the end of the sequence. If no 
events occurred during the capture, the event sequence is sim- 
ply a timed delay and a screen capture. The timed delay is the 
interval between the call to the method startEventSequence and 
the method stopEventSequence. If the method stopEventSequence 

10 is called without previously calling the method startEventSe- 
quence, an EventSequence object is returned, which is essen- 
tially empty, and only contains the current contents of the 
captured screen. 

[0066] A method updateScreenForSequence updates the 

15 screen capture for a given event sequence. Over time, the 
screen captures in a stored event sequence may become outdated. 
This method executes the event sequence, captures the resulting 
screen, and returns the EventSequence object, which now in- 
cludes the updated screen information. 

2 0 Operation . 

[0067] Reference is now made to Fig. 3, which is a flow 

chart illustrating a method of automated verification of user 
interface tests in accordance with a disclosed embodiment of 
the invention. For clarity, the method is described with refer- 
25 ence to the system 10 (Fig. 1) and a modification of the Java 
Device Test Suite. However, the method disclosed with reference 
to Fig. 3 is by no means limited to use with the system 10, and 
may be practiced with other test frameworks and system archi- 
tectures . 
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[0068] At initial step 42 a suite of tests is config- 

ured conventionally. It is assumed that the tests are to be run 
using a user interface, and that interactions between the 
user 18 and the target devices 12 are conducted using the dis- 
5 play 22 (Fig. 1) . The details of preparing, loading, and se- 
lecting tests are outside the scope of the invention, and are 
therefore not discussed. However, it is assumed that the tests 
have been debugged and verified, and are in condition for use 
in evaluating target devices. It should be noted that certain 

10 types of tests are not currently suitable for automatic verifi- 
cation, for example, tests involving such controls as tickers 
and dates. The test framework 26 is configured. Exemplary con- 
figuration details for the Java Device Test Suite (ver. 1.0) 
are given in Appendix 1. It is recommended to define a re- 

15 cording session when using the test framework 26. 

[0069] Next, at step 44, a first phase of operation is 

begun, in which a test is selected from the suite of tests con- 
figured in initial step 42. 

[0070] Control now passes to step 46, where the test 

20 selected in step 44 is executed in manual recording mode. In 
current embodiments of the system 10 (Fig. 1), a test descrip- 
tion window is loaded on the server side, which describes the 
test along with a control panel. The control panel buttons al- 
low the user 18 to control test execution flow. The control 

25 buttons are labeled '^Start Record", ^^Capture Screen", ^^Stop", 
^""Passed" and Failed". The user 18 is required to verify that 
the test running on the system 10 is indeed performing accord- 
ing to specifications. As the tests were all previously veri- 
fied in initial step 42, it is expected that they will also 

30 pass in step 46. The actions of the control buttons are briefly 
described. 
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[0071] By clicking on a ^^start record" button, the re- 

cording process is begun. From this point forward in time, 
every action taken by the user 18 on the actual device, which 
may be shown on the display 24, is recorded and saved, for ex- 
5 ample, scrolling or clicking on a command control. 

[0072] During the recording process the user typically 

performs certain actions that change the appearance of the cur- 
rent test's screen, e.g., insertion of characters into a text- 
box, or selection of an item displayed on a list box. These ac- 

10 tions and responses of the reference target device are examples 
of events of the test program. Screens resulting from the 
events are generally captured and saved. Later, when executing 
in an automation verification mode, the actions that the 
user 18 performed are replayed by an emulator, and screen im- 

15 ages produced by the user and the emulator are compared. 

[0073] Upon completion of the test a button labeled 

"Passed'' is used by the user 18 to certify that the test was 
performed correctly. It is not necessary to expressly capture 
the final screen, as this occurs automatically. Furthermore, 

20 all recorded events and screens are collected and stored on the 
file system of the test server 14 for later use in automation 
mode . 

[0074] The time interval between events occurring dur- 

ing the test is automatically recorded. It is possible for the 

25 user 18 to cause the test to pause, and the ongoing counting of 
inter-event time to cease. Resumption of the test, including 
inter-event time counting may be resumed at the command of the 
user 18. These operations are accomplished using a ^^Stop but- 
ton", and a Resume button", respectively. 

30 [0075] In the event that the test was not performed 

satisfactorily, or if an error has occurred during the re- 



P9340 



47856 



Ver. 47856S6.doc 



22 

cording process, clicking on a Failed button" at any point 
during the test cancels the procedure, and causes all informa- 
tion that has been accumulated up to that point to be disre- 
garded. 

5 [0076] Upon completion of step 4 6, control passes to 

decision step 48, where a determination is made if more tests 
in the suite of tests that was configured at initial step 42 
remain to be executed and recorded. If the determination at de- 
cision step 48 is affirmative, then control returns to step 44. 

10 Otherwise, control proceeds to final step 50. 

[0077] Reference is now made to Fig. 4, which is a flow 

diagram illustrating a second phase of operation, in which the 
suite of tests that was configured at initial step 42 (Fig. 3) 
is replayed, this time automatically by the test server 14 

15 (Fig 1) . The execution of the suite of tests is described with 
reference to one target device, but in practice a plurality of 
target devices 12 may be tested sequentially, or concurrently 
by the test server 14. - At initial step 52, a test is selected 
from the suite of tests configured in initial step 42 {Fig. 3) , 

20 and downloaded to one or more of the target devices 12. All or 
part of the sequence beginning with initial step 52 may be per- 
formed automatically and substantially without human interven- 
tion. 

[0078] Control now passes to step 54, where the test 

25 that was selected in initial step 52 is now reexecuted. The 
test server supplies the event sequence that was recorded in 
step 4 6 to the target device. The event sequence is then re- 
played in the target device. Execution of the supplied event 
sequence causes the target device to mimic the actions initi- 
30 ated by the human operator in step 46 faithfully. During re- 
play, when a screen is generated at the target device, or in an 



P9340 



47856 



Ver. 47856S6.doc 



23 

emulator, it is compared with the corresponding screen of the 
previously recorded event sequence. Thus, the replay process 
returns results associated with the same events as previously 
recorded in step 46. Each screen that was captured in step 46 
5 is compared with a corresponding screen that that was captured 
in the replayed event sequence of step 54, and the results of 
the comparison memorized in a results log file. It should be 
noted that there is no need to re-record the complete test se- 
quence in step 54. Only newly generated screens need to be 
10 evaluated. 

[0079] Next, at decision step 56, it is determined 

whether the results obtained in step 54 match corresponding re- 
sults obtained in step 46. 

[0080] If the determination at decision step 56 is af- 

15 firmative, then control proceeds to step 58, where it is re- 
corded that the test that was replayed in step 54 passed. 

[0081] If the determination at decision step 56 is 

negative, then control proceeds to step 60, where it is re- 
corded that the test that was replayed in step 54 failed. 

20 [0082] Following performance of either step 58 or 

step 60, control passes to decision step 62, where a determina- 
tion is made if more tests in the suite of tests that was con- 
figured at initial step 42 remain to be replayed. If the deter- 
mination at decision step 62 is affirmative, then control re- 

25 turns to initial step 52. Otherwise, control proceeds to final 
step 64, and the procedure ends. 

Upda'blng. 

[0083] Upon completion of a test execution in recording 

mode (step 46, Fig. 3), arrays of bytes are created, each of 
30 which represents an element occurring in the flow of UI test 
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execution. Every recorded event and every captured screen is 
represented as a byte array. While recording of UI tests should 
occur only when it is believed the implementation is stable, 
nevertheless, relatively small changes might still be necessary 
5 to perfect the UI implementation. Such changes would of course 
affect the automatic verification. In order to avoid re- 
recording the test, a facility for updating the record is pro- 
vided. 

[0084] Reference is now made to Fig. 5, which is a 

10 fragmentary schematic illustration of a series of byte ar- 
rays 66 representing a test record from an execution in re- 
cording mode as described with reference to Fig. 3 in accor- 
dance with a disclosed embodiment of the invention. The record 
shows two events, event 1 and event2, interspersed between two 

15 screens, screenl, and screen2 68. All the events and screens 
are represented by byte sequences, shown arbitrarily in upper 
case letters. In particular, screen2 68 has a terminal sequence 
of bytes 70, which corresponds to an icon on the captured 
screen. Assume that it has been necessary to modify the icon. 

20 [0085] Reference is now made to Fig. 6, which is a 

fragmentary schematic illustration of a series of byte ar- 
rays 72, which is identical to the byte arrays 66 (Fig. 5) , ex- 
cept that the screen screen2 68 has been replaced by a screen 
screen2 74 in an updating mode of operation, and which was in- 

25 dividually captured in the same manner as the screen 
screen2 68. The screen Screen2 74 has a terminal sequence of 
bytes 76, which differ from the bytes 70 (Fig. 5) , in that they 
represent a different icon than that of the screen screen2 68. 
The byte arrays 72 may be used in replaying the test that they 

30 represent in the same manner as the byte arrays 66, using the 
procedure detailed above with respect to Fig. 3. 
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[0086] Reference is now made to Fig. 1, which is a flow 

chart illustrating a method of updating a test record in accor- 
dance with a disclosed embodiment of the invention. The process 
begins at initial step 78 where a modification of an existing 
5 test is made. The modification is typically a change in a 
screen, but could be a variation in a user command sequence. 
The previously recorded record of the test is made available. 

[00871 At step 80 an element of the test is performed. 

This may be a screen capture or a user command, such as a mouse 

10 click, text entry, or the like. It should be noted that there 
is no need for re-recording the test sequence during updating. 
The update process is totally automated. The test server sup- 
plies the event sequence that was recorded in step 4 6 (Fig. 3) 
to the target device, which then replays the events. As ex- 

15 plained below, when the replay process reaches a screen needing 
to be updated, it replaces the pre-existing screen with a new 
screen, which is displayed as a consequence of replaying the 
events. The replay process returns a new event sequence with 
the same events but with new captures of the screens. 

20 [0088] Next, at step 82, a new byte array is generated, 

representing a screen resulting from the element replayed in 
step 80. 

[0089] Control now proceeds to decision step 84. A de- 

termination is made whether more elements of the test need to 
25 be executed. It is often not necessary to perform the entire 
test. Once it is recognized that all modified elements have 
been processed, and their respective byte arrays newly stored 
in the test record, the test may be discontinued. 

[0090] If the determination at decision step 84 is af- 

30 firmative, then control returns to step 80. 



P9340 



47856 



Ver. 47856S6.doc 



26 



[0091] 



If the determination at decision step 84 is 



negative, then control proceeds to final step 86, and the pro- 
cedure ends. 



5 art that the present invention is not limited to what has been 
particularly shown and described hereinabove. Rather, the scope 
of the present invention includes both combinations and sub- 
combinations of the various features described hereinabove, as 
well as variations and modifications thereof that are not in 
10 the prior art, which would occur to persons skilled in the art 
upon reading the foregoing description. 

COMPUTER PROGRAM LISTINGS 



[0092] 



It will be appreciated by persons skilled in the 



Listing 1 



20 



15 



String n = 

Configuration . getProperty 

( "com . sun . Icdui . eventHandler " ) ; 
if (n == null) { 

if (Configuration . getProperty 
( "microedition . configuration" ) != null) { 

n = "com . sun .midp . Icdui . Def aultEventHandler " ; 
} else { 



n = "com . sun . midp . Icdui . AWTEventHandler " ; 

1 



25 



Class c = Class . forName(n) ; 

return ( EventHandler ) c . newlnstance ( ) ; 



APPENDICES 



Appendix 1 . 



30 



MIDP Agent Configuration. 

Open the internal . configuration file that is found under 
the lib directory. The value of the 

' com . sun . midp . Icdui . eventHandler ' attribute must be 



P9340 



47856 



Ver. 47856S6.doc 



27 

changed from "com . sun .midp . Icdui . Def aultEventHandler 
to "com . sun .midp . Icdui . AutomationEventHandler " . 
The recommended situation of the edited file is : 
#com . sun . midp . Icdui . eventHandler : 
5 com . sun . midp . Icdui . Def aultEventHandler 

com . sun . midp . Icdui . eventHandler : 
com . sun . midp . Icdui . AutomationEventHandler 
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